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Introduction 

The federal No Child Left Behind Act (NCLB, P.L. 107-110) sparked a national 
approach to standards-based accountability that built on (and, at times, around) the 
actions individual states had been pursuing to varying extents for much of the previous 
decade (Mintrop, 2004). NCLB called for annual testing of all students in grades three 
through eight, a twelve-year timeline to achieve universal proficiency in reading and 
mathematics, identification of low-performing schools based on aggregated and 
disaggregated test scores, and administration of severe consequences for low-performing 
schools that fail to improve. Under the “new accountability” system (Fuhrman, 1999), 
schools are accountable to higher levels of the educational system (school districts or 
state education agencies), but are held accountable for producing specific levels or 
improvements in student learning outcomes, not for delivering designated educational 
inputs and processes (O’Day, 2002). 

Since NCLB’s passage four years ago, states have invested time, personnel, and 
considerable financial resources to develop and implement accountability systems that 
meet the law’s requirements. As these accountability systems have been put into place 
and as increasing numbers of schools have been identified as “in need of improvement” 
(or some other designation to indicate a failure to meet externally mandated performance 
gains), the capacity of state education agencies (SEAs) to shift their responsibilities from 
compliance to intervention systems has been called into sharp relief. There are valid 
reasons to question SEAs’ collective capacity to respond to increased accountability 
requirements. Over a decade ago, states’ responses to other attempts to foster systemic 
reform — the Goals 2000: Educate America Act (1994) and the 1994 Improving 
America’s Schools Act (IAS A), the reauthorization of Title I, Part A — revealed 
widespread difficulties states encountered in implementing fully the accountability 
provisions of Title I and attaining policy cohesiveness (GAO, 2004; Riddle, 2005; 
Superfine, 2005). The earlier standards-based reform efforts revealed state education 
agencies’ deficiencies in terms of the instructional expertise necessary to make a 
difference in troubled schools, the remedies that were put together to support low- 
performing schools, and the amount of assistance that states had budgeted to support the 
remedy or sanction side of accountability (Fuhrman, 1999). 

With the new and heightened accountability requirements NCLB brings, states have the 
responsibility of labeling schools’ (and districts’) performance using a set academic 
standards, numeric academic and non-academic indicators. Along with this responsibility 
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comes the obligation of ensuring that all students have equitable opportunities to meet 
established performance standards, as well as of supporting teachers and administrators in 
struggling schools (Rennie Center, 2005). 



SEAs have put into place a range of strategies and technical assistance programs that 
target schools identified as in need of improvement (Mintrop, 2003; U.S. Department of 
Education, 2004). Ideally, the reams of accountability data now available should make it 
possible for education leaders to determine the quality of such strategies and 
interventions (Doran, 2003). To shed light on how states can use school performance 
data to support failing schools, the present study examined a subset of underperforming 
schools in South Carolina in the wake of a series of state- supported interventions. 1 The 
study, part of a comprehensive evaluation of the South Carolina External Review Team 
Program, made extensive use of annual student performance data collected from schools 
in need of improvement, information that conceivably could be used to guide decisions 
for school and Program improvements. 

Background: South Carolina’s External Review Team Program 

In South Carolina, the state’s Education Accountability Act of 1998 (EAA) brought 
together provisions ranging from the development of detailed, grade level, specific 
academic standards in core content areas, the administration of annual standards-based 
assessments in grades 3 through 8, and steps to be taken for intervention in low- 
performing schools. An important outcome of the EAA was the shift in the direction of 
interventions: rather than focusing on districts (as the state had under its 1984 Education 
Improvement Act), EAA called for schools to be the target of technical assistance. This 
shift required that the state develop appropriate school-level technical assistance and the 
capacity to serve a greater number of entities. One component in South Carolina’s 
arsenal of intervention strategies is the participation of external review teams (ERTs). In 
place since 2000, ERTs are small groups of skilled and experienced individuals who 
work closely with identified schools’ staff, central offices, and local boards of trustees in 
the design of the school's plan, implementation strategies, and professional development 
training to improve student performance and increase the rate of student progress. 

ERTs are assigned to all schools that receive an “unsatisfactory” academic performance 
rating, the lowest of the five categories on the EAA’s school performance classification 
system. Additionally, schools that receive a “below average” performance rating (the 
next to the lowest category) may request an ERT visit. The ERT review is one in a series 
of related school improvement activities coordinated between the school and the South 
Carolina Department of Education, which is responsible for providing continuous follow- 
up and support activities to the underperforming school, including professional 
development, financial support, and on-site assistance. 



1 The work reported here was part of a larger evaluation study of the External Review Team Program 
funded by the South Carolina Education Oversight Committee (EOC) and completed in January, 2006. 
Further details can be found at http://www.sceoc.org/ 
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The ERT review process is designed to diagnose areas of strength and weaknesses. A 
chairperson and a coordinator (a local contact person who is often the principal of the 
school under review) guide each ERT. Prior to conducting the on-site visit, all ERT 
members participate in a comprehensive training session: a one-day refresher session for 
veteran ERT members, with an additional follow-up training session for new ERT 
members. ERT members receive a stipend for their service. 

ERT members spend a minimum of four consecutive days visiting the school, reviewing 
documents of student performance, attendance rates, and other pertinent data both before 
and during the on-site review, conducting interviews of parents, teachers, students, and 
principals, and observing every teacher. The Office of School Quality provides 
documents that guide the on-site data collection and analysis. Reviews are conducted in 
the fall and are concluded before the school’s winter break. 

At the conclusion of each review, the ERT compiles evidence-based recommendations, 
which are shared with the principal and are then assembled into a final report. After the 
approval of the recommendations by the State Board of Education, the Department of 
Education delineates the activities, support, services, and technical assistance it will 
provide to the school. With the approval of the state board, this assistance continues for at 
least three years, or as deemed necessary by the review committee to sustain 
improvement. 

Methods 

The current study took place as part of an evaluation of South Carolina’s External 
Review Team (ERT) Program that was executed from April, 2005 to January, 2006. At 
the time the evaluation took place, approximately 80 schools had participated in the ERT 
Program since its inception in the 2001-02 school year and had put into place a variety of 
targeted interventions to support their academic success. The core question guiding the 
evaluation team centered on to what extent the ERT Program and related, subsequent 
interventions actually contributed to academic improvement? Or as Parrish et al. (2005) 
inquire, have the ERT Program’s mechanisms as currently implemented been effective in 
promoting academic success? An ancillary question that stemmed from the evaluation 
activities was, to what extent can student accountability data be used effectively to assess 
Program impact? 

From the onset of the study, the evaluation team realized that attributing the impact of the 
ERT Program to school improvement was, at best, a convoluted effort. As Hanushek and 
Raymond (2003, 2005) point out, assessing the impact of state accountability activities is 
anything but straightforward. For example, there is no consensus regarding how much 
time is required for change to become manifest in schools. Moreover, school 
performance does not typically improve in even and measurable increments, which 
frustrates attempts to detect progress (Elmore, 2003). 

With the inherent limitations of the ERT Program’ s model and implementation in mind, 
the evaluation team chose to focus one of its activities on an analysis of school 



3 




improvement data on a subset of 62 schools that were originally visited in the 2001-02 
school year (schools we termed “Cohort I”). The rationale underlying this decision was 
that this subset of schools have had the longest amount of time to develop and put into 
place the improvement strategies and actions that were originally identified through the 
ERT visit. For Cohort I schools, the evaluation team examined schools’ absolute 
performance ratings in both mathematics and English language arts (and, for high 
schools, the percentage of students eligible for LIFE scholarships and graduation rates) 
for each school year between 2001 and 2005. We then conducted a detailed analysis of 
the recommendations made in ERT reports and carried out follow up interviews with 
ERT members and school teams at 17 of the 62 Cohort I schools to learn more about the 
implementation of the ERTs’ recommendations. 

Findings 

Analysis of student performance data 

Using student performance data gathered from the 62 Cohort I schools’ 2001, 2002, 

2003, and 2004 school report cards, we characterized schools in one of three ways: as 
having Improved to satisfactory status and remained satisfactory, as showing Continued 
Unsatisfactory status, or as showing No Perceived Pattern. Using this system, we found 
that about 7 in 10 of the schools that were originally reviewed in the 2001-02 school year 
demonstrated some ki nd of discemable pattern (Table 1). 

Table 1. Cohort I Schools’ Improvement Patterns 





improved 


Continued 

Unsatisfactory 


No 

Pattern 


Total 


Elementary Schools 


6 


0 


1 


7 


Middle Schools 


9 


11 


5 


25 


High Schools 


12 


5 


13 


30 


Total 


27 


16 


19 


62 



We then focused on those 43 schools for which we could distinguish a performance 
pattern. Beginning with data gathered during 2001, we examined the performance of the 
27 Improved and 16 Continued Unsatisfactory schools in the 3 years that followed their 
initial ERT review. We found that the greatest gains in absolute performance ratings 
took place between 2003 and 2004, two years following the initial review (Figure 1). 
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Figure 1: Absolute performance ratings in schools following ERT visits 
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Some Continued Unsatisfactory schools showed greater improvements in their absolute 
performance ratings between 2001 and 2004 than some of the Improved schools, though 
they were unable to meet the threshold for satisfactory school performance. Since 
different methods are used to determine absolute performance ratings by school level, we 
break out comparisons by elementary, middle, and high schools. 

1. Improved elementary and middle schools 

All 6 Improved elementary and middle schools entered the ERT program with absolute 
performance ratings that closely bordered the threshold for a satisfactory designation. 
With only one or two tenths of a point away from a satisfactory designation in most 
cases, all Improved elementary schools achieved a satisfactory absolute performance 
rating in the year immediately following the initial ERT visit. One of these elementary 
schools improved beyond the level of “below average” to “average.” By contrast, none 
of the 9 Improved middle schools achieved beyond the rating of “below average,” despite 
sizeable gains in performance. The greatest gains in elementary and middle schools’ 
absolute performance ratings took place between the 2003 and 2004 report card years 
(Figures 2 and 3). 

Figure 2: Absolute performance ratings in Improved elementary schools 
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Figure 3: Absolute performance ratings in Improved middle schools 
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ELA performance among Improved elementary and middle schools showed the greatest 
gains in student performance between 2003 and 2004, whereas sizeable gains in math 
performance were made every year except for 2004. 



2. Continued Unsatisfactory elementary and middle schools 

No elementary school reviewed in 2001-02 fell into the Continued Unsatisfactory 
category. Eleven middle schools, however, demonstrated Continued Unsatisfactory 
performance following the 2001-02 ERT visit. These 11 middle schools demonstrated 
inconsistent patterns in student performance. For example, the greatest number and gains 
in absolute performance ratings took place in the year immediately following their ERT 
visit, when ERT recommendations had yet to be implemented. Absolute performance 
ratings either remained the same or showed only slight changes during subsequent years, 
when these middle schools had more time to implement recommendations and deploy 
technical assistance (Figure 4). 



Figure 4: Absolute performance ratings in Continued Unsatisfactory middle 
schools 
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3. Improved and Continued Unsatisfactory high schools 

Longitudinal changes in high school performance are more difficult to map than changes 
in elementary and middle schools. Whereas elementary and middle school performance 
is determined using student achievement on PACT scores, several performance measures 
determine a high school’s performance rating, including student proficiency on the 
BSAP/HSAP exit exams, eligibility for LIFE scholarships, exit exam passage rates, and, 
after 2003, graduation rates. Sizeable improvements in any one of these areas, or 
cumulative smaller improvements, can result in considerably larger high school 
performance gains. 

Improved high schools 

More of the high schools from the 2001 cohort improved their performance ratings 
during the course of the ERT Program than elementary or middle schools (Figure 4 
above). Further, despite earning the lowest absolute performance ratings of any school 
type during the first year of the ERT Program, Improved high schools showed the highest 
absolute performance ratings and the most improvement between 2001 and 2004. 
Improved high schools gained an average of 1.2 index points, from an average absolute 
performance rating of 1.9 (“unsatisfactory”) in 2001 to 3.1 (“good”) in 2004 (Figure 5). 

Figure 5: Absolute performance ratings in Improved high schools 
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A word of caution should be inserted here. Sizeable gains in high school performance 
may be attributable to ERT processes and recommendations, or to comprehensive efforts 
to modify curricula, leadership and governance, and/or professional development. 
However, changes in the BSAP/HSAP exit exam administration and scoring, as well as 
the 2003 addition of graduation rate as a performance measure, suggest that this trend 
may also reflect modifications in the calculation of high school performance. Our 
analyses suggest that student proficiency on the ELA and math components of the 
BSAP/HSAP exit exams among Improved high schools showed no coherent pattern. 
Similarly, Improved high schools showed only marginal aggregate gains in tenth grade 
BSAP/HSAP passage rates, eligibility for LIFE scholarships, and graduation rates 
between 2001 and 2004. 
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Continued Unsatisfactory high schools 

Continued Unsatisfactory high schools showed no coherent pattern in absolute 
performance ratings. Longitudinal ELA and math proficiency among Continued 
Unsatisfactory high schools showed inconsistent patterns between 2001 and 2004. 
However, the greatest gains in ELA proficiency and the most consistent gains in both 
ELA and math proficiency tenth grade passage rates on the BSAP/HSAP exit exam, and 
graduation rates (Table 2) took place between 2003 and 2004. These trends are 
consistent with the notion that there is a time lag between the implementation of ERT 
recommendations and measurable outcomes. 

Table 2. Graduation rates in improved and Continued Unsatisfactory 
high schools 





Improved 


Continued 

Unsatisfactory 


2003 


74% 


51% 


2004 


75% 


55% 



Analysis of ERT recommendations 

To supplement the analysis of student performance data, we examined the ERT reports 
for 17 Cohort I schools (Table 3). Our analysis focused on the indicators that ERTs used 
in their reporting and on the technical assistance that was identified to remedy 
performance gaps. We were interested in determining whether different types of 
identified deficiencies exist across schools that have improved relative to those that 
remain unsatisfactory, and whether there were different patterns in the technical 
assistance that was deployed to remedy the observed problems. 

Table 3. Schools included in the analysis of ERT recommendations 



Improved 

Schools 


2001 

Rating 


Continued Unsatisfactory 
Schools 


2001 

Rating 


Ridge Spring-Monetta High 


1.6 


Baptist Hill High 


1.0 


C E Murray High 


1.6 


Whitlock Junior High 


1.9 


Lake City High 


2.1 


Heyward Gibbes Middle 


2.1 


Luther Vaughan Elementary 


2.1 


Allendale-Fairfax Middle 


2.1 


Marlboro County High 


2.0 


Fairfield Middle 


2.1 


Macedonia Elementary 


2.1 


Eau Claire High 


2.0 


Mary Ford Elementary 


2.1 


C A Johnson High 


1.4 


West Hardeeville Elementary 


2.0 


Bennettsville Middle 


2.1 


J. V. Martin Jr. High 


2.1 





ERTs evaluate unsatisfactory schools using indicators clustered in four focus areas, 
categories that also organize ERTs’ findings and recommendations: 

• Leadership and Governance, 

• Curriculum and Instruction, 

• Professional Development, and 

• Performance. 
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